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FOREWORD 


Among  the  responsibilities  assigned  to  the  Office  of  the  Manager,  National 
Communications  System,  is  the  management  of  the  Federal  Telecommunication 
Standards  Program.  Under  this  program,  the  NCS,  with  the  assistance  of  the 
Federal  Telecommunication  Standards  Committee  Identifies,  develops,  and 
coordinates  proposed  Federal  Standards  which  either  contribute  to  the  Inter¬ 
operability  of  functionally  similar  Federal  telecommunication  systems  or  to  the 
achievement  of  a  compatible  and  efficient  Interface  between  computer  and 
telecommunication  systems.  In  developing  and  coordinating  these  standards,  a 
considerable  amount  of  effort  Is  expended  In  Initiating  and  pursuing  joint 
standards  development  efforts  with  appropriate  technical  committees  of  the 
Electronic  Industries  Association,  the  American  National  Standards  Institute, 
the  International  Organization  for  Standardization,  and  the  International 
Telegraph  and  Telephone  Consultative  Committee  of  the  International 
Telecommunication  Union.  This  Technical  Information  Bulletin  presents  an 
overview  of  an  effort  which  is  contributing  to  the  development  of  compatible 
Federal,  national,  and  International  standards  In  the  area  of  Video 
Teleconferencing.  It  has  been  prepared  to  inform  Interested  Federal  activities 
of  the  progress  of  these  efforts.  Any  comments.  Inputs  or  statements  of 
requirements  which  could  assist  In  the  advancement  of  this  work  are  welcome  and 
should  be  addressed  to: 


Office  of  the  Manager 
National  Communications  System 
ATTN:  NCS-TS 
Washington,  DC  20305 
(202)  692-2124 
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SECTION  1  INTRODUCTION  AND  SUMMARY 


This  document  summarizes  work  performed  by  Delta  Information 
Systems,  Inc.  for  the  office  of  Technology  and  Standards  of  the 
National  Communications  System,  an  organization  of  the  U.S. 
Government,  under  contract  number  DCA100-83-C-0047  Modification 
P00004.  The  work  was  performed  under  Subtask  1  (Development  of  A 
Video  Tape  to  Test  Teleconferencing  Codecs)  under  Task  3.  The 
Office  of  Technology  and  Standards,  headed  by  National 
Communications  System  Assistant  Manager  Marshall  L.  Cain,  is 
responsible  for  the  management  of  the  Federal  Telecommunications 
Standards  Program,  which  develops  telecommunication  standards 
whose  use  is  mandatory  by  all  Federal  agencies. 

This  report  covers  the  design  and  production  of  a  video  tape 
for  testing  motion  codecs.  It  is  part  of  a  program  for 
comparative  evaluation  of  motion  codecs  used  for  teleconferencing 
at  1.544  Mbs.  The  specific  purpose  of  the  tape  is  to  provide 
pictorial  material  which  may  occur  during  both  government  and 
commercial  teleconferences  and  which  simultaneously  stresses  the 
capabilities  of  each  codec  sufficiently  so  that  differences  in 
performance  will  become  apparent.  The  choice  of  material  must 
lead  to  a  completely  impartial  evaluation  without  favoring  any 
particular  algorithm,  and  must  not  be  constrained  by  any  present 
limitation  of  teleconferencing  which  will  not  exist  in  the  future 

Section  2  of  this  report  describes  the  relevant  background 
material  and  information  which  were  reviewed  before  starting  the 
tape  design.  Subsequently,  the  various  categories  of  picture 
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material  which  were  candidates  for  the  codec  test  tape  are 
analyzed  in  Section  3.  The  multiple  steps  that  were  followed  in 
the  preparation  of  the  tape  are  detailed  in  Section  4.  An 
original  scenario  was  prepared,  a  suitable  studio  selected,  and  a 
large  variety  of  studio  scenes  and  graphic  material  taped, 
followed  by  more  scenes  in  actual  teleconference  settings.  All 
this  material,  together  with  other  tapes  obtained  from  military 
organizations,  was  edited  into  a  final  test  tape  version.  At  the 
same  time,  a  test  signal  tape  containing  mainly  electronically 
generated  signals  was  prepared.  Section  5  summarizes  reviews  of 
the  tape  by  many  users.  Section  6  contains  the  exact  scenario  of 
both  tapes  in  a  simplified  tabular  format.  Section  7  gives  a 
brief  conclusion  and  recommendations  for  further  efforts. 


SECTION  2  BACKGROUND  MATERIAL 


2.1  Related  Tasks 

The  contents  and  format  of  the  test  tape  which  forms  the  main 
output  of  this  task  are  closely  related  to  two  other  tasks  of 
Contract  DCA100-83-C-0047 .  They  are: 

o  Development  of  a  methodology  to  test  video  codecs  used  in 
teleconferencing 

o  Test  and  evaluation  of  teleconferencing  video  codecs 
transmitting  at  1.5  Mbps 

The  test  methodology  not  only  calls  for  specific  test  signals 
and  sequences  to  be  included  in  the  test  tape  but  also  determines 
its  approximate  length  and  the  overall  layout  of  i*-*  contents. 
Since  this  task  is  being  performed  concurrently  with  the  design  of 
the  test  tape,  the  necessary  exchange  of  information  was  easily 
accomplished.  On  the  other  hand,  the  performance  of  the  codec 
tests  is  still  in  the  planning  stage;  however,  some  fundamental 
ideas  have  already  been  developed,  and  the  test  tape  contents  have 
been  designed  to  satisfy  all  presently  anticipated  test  and 
evaluation  requirements. 

2.2  CCITT/CCIR  Material 

A  number  of  pertinent  documents  were  reviewed  for  guidance  in 
the  task  of  test  tape  preparation.  The  one  most  directly 
applicable  is  in  CCITT  Draft  Recommendation  H.100  covering  the 
split  screen  presentation  of  two  sections  of  a  conference.  It  is 
contained  in  Item  3  of  Contribution  No.  287  of  CCITT  Study  Group 


XV,  dated  March  1984.  This  method  allows  the  efficient 
presentation  of  a  common  conference  situation  with  best  possible 
picture  quality  and  therefore  must  be  included  in  a  good  test 
tape . 

The  basic  document  covering  evaluation  of  TV  picture  quality 
is  CCIR  REC  500-2,  Method  for  the  Subjective  Assessment  of  the 
Quality  of  Television  Pictures  (Vol.  XI,  Part  1,  XVth  Plenery 
Assembly,  Geneva  1982).  Though  intended  for  entertainment  TV,  it 
contains  many  statements  directly  applicable  to  the  codec  test 
tape  and  its  application.  They  are: 

o  Test  pictures  should  be  more  critical  th<-n  normal  pictures 
o  Observers  should  be  well  briefed  and  introduced  to  likely 
impairments . 

o  Between  10  and  20  observers  should  be  used  for  picture 
quality  evaluation. 

o  Scales  for  both  absolute  ratings  of  picture  impairment  and 
comparison  of  two  impaired  pictures  are  given, 
o  Viewing  conditions  for  picture  evaluation  are  given. 

A  document  entitled:  Method  for  the  Subjective  Assessment  of 
Visual  Telephone  Picture  Quality  was  prepared  by  CEPT  (Consortium 
of  European  PTT's),  COST  211  Committee  dealing  with  visual 
telephony.  No.  211/82/5,  dated  3-31-1982.  It  largely  re-states 
the  salient  points  of  the  above  referenced  CCIR  Rec .  500-2.  In 

addition,  it  contains  recommendations  for  the  type  of  test 
pictures,  the  length  of  the  test  scenes,  and  most  pictures,  the 
length  of  the  test  scenes,  and  most  important,  for  a  important. 


SECTION  3  TAPE  REQUIREMENTS 

3.1  General 

All  codecs  inherently  produce  a  deteriorated  output  picture. 
The  deteriorations  can  be  broadly  separated  into  two  catagories; 
picture  quality  and  motion  rendition.  Loss  of  picture  quality 
typically  manifests  itself  by  one  or  all  of  the  following:  Reduced 
resolution,  poor  color  rendition,  spurious  patterns  or  artifacts. 
Deterioration  of  motion  performance  may  show  up  as  jerkiness, 
blurring,  or  partial  or  complete  picture  break  up. 

Consistent  with  the  general  philosophy  of  testing,  the 
picture  material  should  stress  the  codec  performance  close  to  or 
somewhat  beyond  its  limits.  This  is  necessary  to  accentuate  the 
often  small  differences  between  codecs.  A  high  quality  analog 
input  picture  is  required;  indeed,  the  analog  portion  of  the  test 
set  up  must  be  essentially  transparent.  That  makes  it  mandatory 
that  only  1"  tape  be  used  for  the  test  tape  and  for  recording  the 
codec  output.  Any  failure  to  follow  these  rules  will  make  it 
impossible  to  achieve  significant  and  impartial  results. 

The  imposition  of  stress  on  the  codec  appears  to  make 
artificial  scenes  such  as  a  rotating  disk  or  pendulum  with  various 
color  and  black/white  patterns  very  desirable.  Such  patterns  can 
produce  a  stronger  and  more  measurable  stress  than  ordinary  scenes 
and  thus  help  in  identifying  differences  in  codec  performance. 
However,  the  variety  and  complexity  of  codec  algorithms  makes  it 
impossible  at  this  time  to  assume  correlation  between  performance 
with  natural  and  artificial  scenes.  In  other  words,  in  a 


comparison  of  two  codecs  one  unit  may  show  worse  deterioration 
with  an  artificial  scene  but  perform  better  with  a  natural 
conference  scene.  Thus  it  is  possible  for  the  use  of  artificial 
scenes  to  preclude  an  impartial  evaluation.  This,  however,  does 
not  rule  out  that  in  the  future  artificial  scenes  or  patterns  will 
prove  to  be  valuable  tools  for  codec  performance  testing. 

The  subsequent  paragraphs  briefly  discuss  the  types  of 
picture  material ' which  were  considered  for  inclusion  in  the  codec 
test  tape. 

3.2  Still  Pictures 

Still  pictures  for  the  evaluation  of  overall  picture 
appearance  should  be  high  quality  color  slides  of  portraits  and 
persons  in  typical  surroundings.  Such  pictures  show  up  mainly 
problems  with  color  performance.  Graphics  such  as  print  samples, 
typed  pages,  graphs  and  diagrams  primarily  stress  the  resolution 
capability  of  the  codec  system.  Both  are  vital  for  a  full 
evaluation.  The  picture  material  must  be  such  that  the  analog 
picture  is  of  excellent  quality  and  in  no  way  limited  by,  for 
instance,  the  resolution  capability  of  the  system.  Still  pictures 
should  be  considered  for  special  tests  such  as  transmission  error 
conditions  where  motion  is  not  a  primary  issue  but  may  tend  to 
confuse  the  evaluator. 

3.3  Motion  Scenes 

3.3.1  Simulated  Conference 

Since  the  intended  use  of  the  codecs  to  be  tested  is  for 


teleconferences,  scenes  of  typical  conference  situations  represent 
the  most  important  picture  material  for  the  test  tape.  It  is 
impossible  to  cover  the  gamut  of  possible  conference  scenes. 

Since  teleconferencing  is  relatively  new,  more  applications  are 
likely  to  be  developed  continuously.  Various  government  agencies 
obviously  will  present  different  requirements  and  will  not  wish  to 
be  bound  by  the  constraints  often  imposed  by  commercial 
teleconferencing  operators.  Therefore,  the  conference  scenes 
should  not  be  limited  to  the  conventional  arrangement  of 
participants  sitting  at  a  conference  table  with  a  special  fixed 
layout.  The  conferees  should  also  be  shown  getting  up,  presenting 
materials  at  at  a  board  or  projection  screen,  and  taking  part  in 
lively  discussions.  Camera  panning  and  zooming  should  be  used 
judiciously. 

Most  existing  commercial  teleconference  installations  provide 
the  above  functions  to  a  limited  extent.  Therefore,  exactly 
following  only  established  commercial  procedures  would  fail  to 
produce  a  test  tape  which  subjects  the  codecs  to  the  stress  which 
is  needed  to  highlight  differences  in  their  performance.  Thus  it 
becomes  necessary  to  include  scenes  which  are  staged  to 
intentionally  stress  the  codec  to  and  beyond  its  performance 
limits  which  can  easily  be  done  is  a  TV  studio  setup.  This  is 
also  expected  to  cover  most  situations  likely  to  occur  at  future 
government  agency  conferences. 

3.3.2  Action  Scenes 

It  is  expected  that  during  many  military,  and  also  other 


teleconferences,  reference  will  be  made  to  important  events  which 
have  been  recorded  on  video  tape.  For  discussions  of  these  events 
by  all  teleconference  participants  the  video  tapes  will  have  to  be 
transmitted  via  codecs.  Therefore,  it  is  desirable  to  include 
some  scenes  with  considerable  action  on  the  tape.  Military  scenes 
obviously  are  preferred,  but  others  such  as  sporting  events  need 
not  be  ruled  out.  Caution  is  necessary  in  the  choice  of  action. 
For  instance,  on  automobile  race  taken  from  overhead,  in  spite  of 
very  fast  action,  is  a  poor  choice.  The  racing  cars  kept  close  to 
the  center  of  the  picture  appear  almost  stationary  while  the 
surroundings  move  so  fast  that  they  are  barely  distinguishable 
even  on  an  analog  picture.  Similarly,  a  missile  shortly  after 
launch  against  a  sky  background  does  not  give  relevant  results. 

3.3.3  Computer  Animation 

Computer  generated  animated  picture  material  is  gaining  ever 
increasing  popularity.  It  is  very  useful  not  only  for  commericals 
on  entertainment  TV  but  also  for  impressive  presentation  of  an 
unlimited  variety  of  data.  Examples  are  animated  curves,  bar 
charts,  pie  charts,  maps  or  military  type  displays,  all  featuring 
multi-colored  rapid  motion  displays.  This  material  can  be 
recorded  on  tape  directly  out  of  the  computer. 

3.4  Test  Signals 

3.4.1  Standard  Test  Signals 

Elaborate  methods  have  been  developed  for  end-to-end  testing 
of  analog  TV  systems.  These  techniques  are  directly  applicable  to 


high  quality,  high  bit  rate  digital  TV  systems  which  provide 
broadcast  TV  quality.  But  their  relevance  for  codecs  which  yield 
quality  significanly  less  than  broadcast  TV  remains  to  be 
demontrated.  In  the  related  task  for  the  development  of  test 
methodology,  recommendations  are  made  for  the  performance  of  a 
selected  group  of  analog  tests  on  codecs.  But  the  usefulness  of 
these  recommendations  remains  to  be  proven.  Obviously  such  test 
signals  would  be  evaluated  objectively  with  test  instruments. 
Therefore,  they  should  not  be  included  in  a  test  tape  designed  for 
subjective  evaluation  by  a  panel  of  observers  but  must  be  handled 
separately.  However,  it  would  be  extremely  valuable  if 
correlation  could  be  established  between  the  subjective  and 
objective  tests  because  it  would  lead  to  a  much  less  cumbersome 
and  costly  codec  evaluation  technique. 

3.4.2  Artificial  Motion 

Many  test  signals  can  be  generated  electronically  by  means  of 
conventional  switching,  special  effects  and  test  equipment  to 
measure  the  ability  of  the  codec  to  reproduce  motion.  Measured 
amounts  of  pixel  change  between  frames  can  be  produced.  It  is 
also  possible  to  devise  instrumentation  to  perform  limited 
objective  motion  measurements.  However,  since  correlation  between 
artificial  and  natural  motion  has  not  been  established,  such 
signals,  too,  should  not  be  included  in  a  test  tape  designed  for 
subjective  evaluation  but  handled  separately  in  further  studies. 


SECTION  4  TAPE  PREPARATION 


4.1  Studio  Tape  Scenario 

After  thorough  review  of  the  requirments  summarized  in 
Section  3,  an  initial  scenario  for  the  studio  portion  of  the  test 
tape  was  developed.  It  was  recognized  that  the  resulting  tape 
would  not  be  the  final  version.  However,  this  scenario  was  made 
up  to  include,  if  possible,  all  scenes  that  required  the  use  of 
one  or  more  live  TV  cameras.  The  use  of  a  studio  facility 
requires  considerable  lead  time  and  is  quite  costly;  therefore, 
proper  planning  at  this  point  saves  both  time  and  expenditure. 

The  initial  studio  scenario  was  made  up  with  the  objective  of 
an  ultimate  length  of  up  to  20  minutes.  Its  contents  were  as 
follows : 

(A)  Introduction 

This  part  consists  of  a  title  and  a  test  signal  for 
calibration . 

(B)  Simulated  Conference 

A  variety  of  about  12  separate  scenes  was  designed  to  include 
most  activities  expected  to  occur  in  a  conference.  Six 
participants  are  shown  entering  and  leaving,  sitting,  standing  and 
walking.  Shots  of  all  six,  groups  of  three,  and  close  ups  of  a 
single  person  are  included.  Demonstrations  and  discussions  using 
a  marker  board,  viewgraph  slides  and  maps  are  shown.  Some  of  the 
participants  wear  lively  patterned  clothing  and  the  scenes 


generally  include  a  lot  of  motion. 

(C)  Still  Pictures 

A  selection  of  different  types  of  pictures  is  included,  such 
as  standard  SMPTE  (Society  of  Motion  Picture  and  Television 
Engineers)  color  slides,  a  typed  page,  print  samples  in  various 
sizes  and  colors,  a  circuit  schematic,  a  map  with  details  in 
color,  and  an  aerial  photograph. 

(D)  Electronically  Produced  Scenes 

This  section  is  made  up  of  portions  of  existing  tapes  with 
preferably  military  action  and  scenes  generated  by  Dubner  computer 
animation . 

(E)  Conclusion 

This  part  contains  the  end  title  and  a  repeat  of  the 
calibration  test  signal. 

In  addition  to  the  above,  tentative  plans  were  made  for  a 
separate  test  signal  tape  which  would  be  produced  at  the  time  of 
final  editing  and  not  used  for  subjective  evaluation.  This  tape 
contains  selected  standard  analog  TV  test  signals  such  as  video 
sweep,  composite  test  signal  and  modulated  and  unmodulated 
staircase  and  ramp  signals.  Artificial  motion  is  produced  for 
instance  by  switching  between  a  window  signal  and  a  flat  field  and 
by  producing  outlines  of  special  effect  transition  patterns  on  a 
flat  field.  These  outlines  are  then  moved  across  the  screen  in 
both  directions  at  various  speeds.  Examples  are  a  vertical  bar  or 


a  diamond  shaped  pattern  originating  in  the  center  of  the  picture 
The  only  signal  not  generated  electronically  is  a  resolution  test 
chart  which  is  recorded  together  with  the  other  still  pictures  of 
the  regular  tape  but  used  only  on  the  test  signal  tape. 

4.2  Facility  Selection 

Several  firms  were  available  locally  to  produce  video  tapes. 
The  main  business  of  these  firms  is  production  of  TV  commercials 
and  a  large  variety  of  promotion  and  training  material.  They  all 
have  fully  equipped  studios,  control  rooms,  recording  and  editing 
facilities  available.  Particularly  the  large  firms  are  heavily 
oriented  towards  artistic  production.  For  the  special  purpose  of 
the  test  tape  with  minimal  artistic  but  somewhat  non-standard  ' 
technical  requirments,  it  became  apparent  that  a  somewhat  smaller 
firm  would  be  more  flexible  and  responsive.  Thus  Center  City 
Video  was  selected  and  fully  lived  up  to  expectations. 

4.3  Studio  Production 

A  studio  set  was  constructed  for  the  live  scenes  of  the 
simulated  conference.  It  consisted  of  two  conference  tables  with 
three  chairs  each  forming  an  obtuse  angle.  A  viewgraph  projector 
stood  on  a  pedestal  in  the  center.  A  projection  screen,  maps  and 
a  markerboard  were  mounted  on  the  wall  behind.  The  remainder  of 
the  background  was  covered  with  a  patterned  wallpaper  and  striped 
drapes.  Two  cameras  equipped  with  zoom  lenses  on  pan  and  tilt 
heads  provided  full  flexibility  for  all  desired  shots.  The  still 
pictures  were  subsequently  produced  using  a  single  camera  and 
front  or  back  lighting  or  projection  depending  on  the  format  of 


the  picture  material.  Zooming  on  still  pictures  was  used  in 
selected  instances. 

Six  semi-professional  actors  ("extras")  were  selected  to  be 
the  teleconference  participants.  A  program  director  supplied  by 
Center  City  Video  operated  the  switching  equipment  and  gave 
instructions  to  the  studio  camera  and  video  tape  operators.  The 
program  director  had  an  outline  of  the  desired  scenes  but  specific 
instructions  were  given  by  DIS  personnel  present  in  the  control 
room.  The  scenes  included  the  gamut  from  wide  angle  shots  showing 
all  participants  to  head  and  shoulder  close-ups  of  a  single 
person.  Actions  were  designed  to  cover  all  gestures  likely  to 
occur  during  a  teleconference,  such  as  talking  while  seated, 
presentations  at  a  marker  board,  and  explanations  of  viewgraphs 
and  maps.  Some  fairly  lively  discussions  were  included.  Zooming, 
camera  panning  and  tilting  were  used  as  appropriate. 

4.4  Typical  Teleconference  Material 

Representative  scenes  of  present  commercial  teleconferences 
were  taped  in  several  different  teleconference  rooms.  It  would 
have  been  ideal  to  use  portions  of  actual  teleconferences  but 
company  privacy  considerations  make  this  generally  impossible. 
Therefore,  conferences  using  all  available  facilities  had  to  be 
staged.  The  following  rooms,  each  featuring  a  different  layout, 
were  utilized. 

A.  ATT-PMS 

The  ATT  Picturephone  Meeting  Service  room  has  the  layout 


shown  in  Fig.  4-1.  The  three  face-to-face  cameras  are  fixed  and 
each  provide  a  close-up  of  2  people  sitting  at  the  conference 
table  which  accomodates  6  people.  Switching  between  them  is 
controlled  by  the  microphones,  with  the  conferee  talking  loudest 
shown  on  the  output  picture.  The  overview  camera  shows  a  fixed 
wide-angle  shot  of  the  whole  conference  table.  The  multipurpose 
camera  has  a  remotely  controlled  pan,  tilt  and  zoom  capability 
allowing  any  shot  from  the  whole  room  to  extreme  closeups.  It  can 
be  used  for  following  somebody  walking  but  it  is  mainly  intended 
to  cover  the  marker  board.  A  frosted  glass  panel  which  can  be 
back  lighted  for  transparencies  is  provided  for  graphics  which  are 
shown  by  means  of  an  overhead  camera  with  zoom  capability.  A  35mm 
slide  camera  is  in  the  equipment  room.  The  main  camera  control 
panel  is  in  the  center  of  the  conference  table  convenient  to  the 
conference  moderator.  A  duplicate  control  panel  is  in  a  corner  of 
the  room . 

A  considerable  length  of  typical  shots,  using  all  cameras, 
was  recorded,  including  a  presentation  at  the  marker  board. 
Operation  of  the  multipurpose  and  graphics  cameras,  being 
performed  by  untrained  personnel,  was  somewhat  rough  but  this  is 
typical  for  a  teleconference. 

B.  Satellite  Business  Systems 

The  SBS  teleconference  room,  configured  specifically  for  use 
by  Aetna  Insurance  Company,  is  shown  in  Fig.  4-2.  The  two  fixed 
face-to-face  cameras  each  cover  3  conferees  on  one  side  of  the 
table.  The  "black  board"  camera  shows  mainly  the  material  on  the 


FIG.  4-1  PMS  TELECONFERENCE 
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FIG.  4-2  SDS  TELECONFERENCE  ROOM 


easel  but  has  pan,  tilt  and  zoom  capability  so  that  it  can  be  used 
for  other  purposes.  However,  it  cannot  provide  coverage  of  the 
whole  conference.  The  graphics  camera  can  be  zoomed  and  focused. 

A  slide  camera  is  available  in  the  equipment  room.  SBS  mainly 
uses  the  principle  of  "continuous  presence"  which  multiplexes  the 
two  face-to-face  camera s  and  a  freeze  frame  of  the  graphics 
camera.  Therefore  only  a  limited  switching  capability  has  been 
provided.  For  the  purpose  of  the  test  tape,  only  a  single  picture 
can  be  handled  at  one  time;  thus  the  multiplexing  feature  could 
not  be  used.  Separate  typical  pictures  were  recorded  from  all 
cameras . 

C.  ISACOMM 

The  ISACOMM  teleconferencing  room  is  shown  in  Fig.  4-3. 

Only  a  single  face-to-face  camera  is  used  which  has  a 
pre-programmed  pan  and  zoom  capability  to  produce  a  variety  of 
different  views;  namely  6  people,  the  2  center  conferees,  and  a 
single  person  with  the  possibility  of  an  extreme  close-up.  A 
manual  over-ride  of  these  functions  with  a  "joystick"  controlled 
pan  and  tilt  and  separate  zoom  is  available.  The  speaker's  camera 
has  one  pre-programmed  position  showing  the  marker  board  with  the 
upper  body  of  the  speaker;  but  the  same  manual  override  control 
allows  following  the  speaker  from  his  seat  to  the  board  and  a 
tight  close-up  of  a  portion  of  the  board.  The  graphics  camera  has 
the  conventional  zoom  function  and  also  a  fine  focus  adjustment. 

A  slide  camera  is  provided  in  the  equipment  room.  A  joystick 
controlled  electronic  pointer  can  be  superimposed  on  an  outgoing 
freeze-frame  picture.  All  these  functions  are  software  controlled 
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from  a  very  sophisticated  control  panel  which  is  handled  by  a 
conferee.  A  miniature  monitor  is  available  next  to  it  to  check 
the  operation.  A  computer  terminal  is  used  to  generate  animated 
graphics  displays. 

Typical  pictures  were  recorded  showing  outputs  from  all 
cameras  performing  all  the  above  described  functions.  The  only 
exceptions  were  the  pointer  which  is  limited  to  the  freeze  frame 
mode,  and  unfortunately  the  computer  produced  animated  graphics. 

The  three  facilities  in  which  recordings  were  made  are 
representative  of  suppliers  of  commercial  teleconferencing 
services.  Though  similar  in  their  fundamental  function,  the  three 
rooms  differ  drastically  in  detailed  layout  and  equipment.  Thus 
the  scenes  recorded  in  them  are  representative  of  the  majority  of 
commercial  teleconferences  held  at  this  time.  Undoubtedly,  as 
this  presently  infant  industry  develops  in  the  future,  an  ever 
larger  variety  of  scenes  with  always  lessening  constraints  will  be 
used . 

4.5  Editing 

The  tapes  produced  in  the  studio  and  the  three  teleconference 
rooms  resulted  in  a  large  excess  of  material.  Only  the  most 
significant  scenes  were  retained,  and  even  they  were  drastically 
shortened.  Print  samples  produced  by  a  "Chyron"  character 
generator  showing  different  size  characters  in  all  combinations  of 
primary  and  secondary  colors  were  inserted.  Scenes  of  computer 
generated  animation  were  taken  from  material  available  at  Center 
City  Video.  Three  sequences  of  military  action  were  taken  from 


tapes  obtains  from  the  US  Navy  and  Air  Force.  A  few  scenes  from  a 
tape  provided  by  the  Defense  Communications  Agency  were  added. 

Some  similar  scenes,  mainly  graphics,  were  shortened  and  combined 
in  order  to  limit  the  overall  tape  length. 

All  taping  of  the  simulated  conferences  resulted  in  a 
somewhat  logical  order  of  scenes.  However,  since  a  series  of 
similar  scenes  may  reduce  the  alertness  of  the  evaluators,  it  was 
decided  to  rearrange  all  sequences  into  a  random  order.  Scoring 
intervals  of  10  seconds  are  included  after  each  sequence.  Three 
sequences  featuring  resolution  and  color  rendition  requirements 
are  repeated  at  arbitrary  spots  on  the  tape  to  cross  check  monitor 
performance.  For  viewing  these  sequences  the  connections  to  the 
two  monitors  will  be  reversed  to  check  whether  possible 
differences  in  monitor  performance  may  influence  the  evaluation. 
Only  two  exceptions  from  the  randomization  were  made.  The 
military  scenes  were  kept  together  at  the  end  of  the  tape. 
Following  the  military  scenes,  three  identical  scenes  with 
moderate  motion  recorded  to  be  used  with  a  data  link  simulator 
which  inserts  artificial  errors.  This  provides  the  optional 
feature  to  evaluate  codecs  under  three  levels  of  error  conditions. 

The  sequences  on  the  edited  test  tape  can  be  broadly  put  into 
several  categories.  The  are:  Still  graphics  (color  and  black  and 
white),  graphics  with  animation  (zoom,  pan  or  pointing),  slow, 
moderate  and  lively  motion  with  different  backgrounds,  computer 
animation,  and  military  action.  There  are  no  distinct  dividing 
lines,  some  sequences  can  fit  into  more  than  one  category.  The 
tape  provides  the  gamut  of  present  day  commercial 


SECTION  5  USER  REVIEW 


The  edited  test  tape  represented  the  best  combined  judgement 
of  DIS  personnel.  In  order  to  test  the  validity  of  this 
judgement,  a  3/4"  copy  of  this  tape  was  presented  for  review  to 
teleconferencing  and  visual  communication  experts  in  both 
Government  Agencies  and  commercial  organizations.  Following  is  a 
summary  of  the  reviewing  organizations  and  personnel  together  with 
highlights  of  their  collective  comments. 

The  following  Government  and  associated  personnel  viewed  the 
test  tape: 

Maj.  W.  Benson,  USAF  -  AFSITT 

Mr.  D.  Bodson,  NCS 

Mr.  T.  Dickinson,  USAF  -  AFXOORV 

Mr.  J.  Elwell,  USN-NUSC 

Mr.  J.  Hart,  USAF-AFSC 

Dr.  R.  Heroux,  USN-NUSC 

Mr.  C.  Hollister,  FEMA 

Dr.  C.  Kelley,  DARPA 

M.  R.  Rydel ,  MITRE 

Maj.  P.  Stewart,  USAF-1947HSG/DCOV 
Mr.  F.  Suraci,  DCA-CCSO 

In  addition,  the  tape  was  reviewed  by  groups  of  technical  and 
operational  personnel  of  ISACOMM  and  SBS.  The  comments  furnished 
by  all  the  above  showed  considerable  uniformity  and  are  summarized 


herewith . 


(1)  The  studio  sequences  of  a  simulated  teleconference  are  useful 
to  stress  the  video  codecs.  Lively  motion  and  bright  saturated 
colors  and  desirable.  Both  the  studio  environment  and  scenes  in 
actual  teleconferencing  rooms  are  important. 

(2)  Almost  any  type  material  may  be  transmitted  in  a  military 
or  other  U.S.  Government  teleconference.  More  flexibility  than 
for  commercial  applications  is  needed.  An  example  is  medical 
diagnostic  services  which  now  employ  analog  TV.  Tapes  of  military 
and  other  events  are  likely  to  be  used. 

(3)  Transmission  of  typed  material  is  very  important. 

A  conferee  shown  signing  a  document  provides  instantaneous  and 
automatically  verified  promulgation  of  an  order. 

(4)  Computer  graphics  and  map  displays  are  desirable. 

(5)  Zooming  in  on  a  single  speaker  and  on  graphics  (such  as 
typed  page,  aerial  photo  and  map)  is  a  meaningful  codec  test. 

(6)  The  Chyron  generated  slides  are  very  good  because  they 
allow  evaluation  of  a  wide  range  of  color  combinations. 

(7)  One  purpose  of  the  tape  is  to  test  what  the  codec  can 

do,  not  to  show  what  will  generally  be  done  with  it.  As  such  the 
test  tape  need  not  be  constrained  to  material  encountered  in  a 
conventional  teleconference . 

The  overall  reaction  of  the  reviewers  was  that  the  test  tape 
was  adequate  and  fulfilled  its  intended  requirements. 


SECTION  6  FINAL  TAPE  SCENARIO 
6.1  Subjective  Evaluation  Tape 

The  primary  output  of  this  task  is  the  test  tape  which  will 
be  used  during  the  next  phase  of  this  program  for  subjective 
evaluation  of  codecs  in  accordance  with  the  previously  developed 
test  methodology.  Following  a  color  bar  chart  used  for  equipment 
check  and  a  title  slide,  the  tape  consists  of  49  sequences,  46  of 
which  are  devoted  to  the  evaluation  of  basic  codec  performance 
parameters,  such  as  resolution,  color  fidelity,  motion  rendition, 
and  artifacts  or  spurious  images.  The  last  3  sequences  are  used 
for  evaluation  of  the  effect  of  transmission  errors  and  are 
separated  by  a  20  second  interval  necessary  to  allow  for  test 
equipment  rearrangement.  A  scoring  interval  of  10  seconds  follows 
each  sequence.  The  total  length  of  the  tape  is  about  31  minutes. 
Deducting  the  "inactive"  periods  yields  a  duration  of  about  22.5 
minutes  for  the  49  sequences  to  be  evaluated,  or  an  average  of 
about  27.5  seconds  per  sequence. 

The  tape  scenario  is  shown  on  Table  6-1  in  a  very  concise 
form.  The  tape  material  is  divided  into  4  categories  each  of 
which  contains  2  to  5  features.  It  is  possible  that  a  pertinent 
feature  from  another  category  is  added.  The  absence  of  a  specific 
feature  is  indicated  by  The  source  of  each  sequence  and  its 


duration  are  given.  Necessary  explanations  are  in  the  remarks 
column.  A  legend  explains  the  symbols  used  for  each  category. 


LEGEND  TO  TABLE  6-1 


SOURCE 

A  Air  Force 
C  Center  City  Video 
D  DCA 

E  Center  City  Video 
I  ISACOMM 
N  Navy 
P  PMS 
S  SBS 

ANIMATED  GRAPHICS 
Material 

B  Black  &  White 
C  Color 

Subject  Matter 
H  Hand  Drawing 
M  Map 

I  P  Printed  Material 

T  Typed  Page 

V  Viewgraph 
Motion 

X  Pan  &  Tilt 

Y  Pointing 
Z  Zoom 


STILL  GRAPHICS 
Material 

Studio  B  Black  &  White 

C  Color 

Electronics  Subject  Matter 

D  Diagram 
P  Print  Sample 
S  Slide 
T  Typed  Page 
Graphics  With  People 
Subject  Matter 
B  Marker  Board 
E  Easel 

F"  Circuit  Board 
M  Map 

V  Viewgraph 
Came ra  Motion 
X  Pan  &  Tilt 
Z  Zoom 
Position 
D  Sit 
U  Stand 
W  Walk 
Motion 
A  Average 
L  Lively 


S  Slow 


PEOPLE 


Position 
D  Sit 
U  Stand 
W  Walk 
Motion 
A  Average 
L  Lively 
S  Slow 
Background 
B  Busy 
P  Plain 
Camera  Motion 
X  Pan  £>  Tilt 
Z  Zoom 

All  Categories 


6.2  Test  Signal  Tape 


The  test  signal  tape  is  not  a  primary  output  of  this  task. 

It  was  developed  because  it  required  only  a  negligible  added 
effort;  and,  after  having  been  processed  through  the  codecs,  is 
expected  to  yield  highly  valuable  results  for  future  studies.  The 
processed  test  signal  tape  will  not  be  shown  to  the  jury  of 
evaluators  who  determine  the  official  ranking  of  the  codecs.  Its 
expected  initial  use  will  be  to  determine  what  correlation  can  be 
achieved  between  those  official  subjective  results  and  the 
objective  measurements. 

The  scenario  of  the  test  signal  tape  is  shown  on  Table  6-2. 

It  contains  the  standard  test  signals  suggested  in  the  test 
methodology  task  for  objective  measurement  of  the  pertinent  analog 
performance  parameters  and  the  signals  which  may  lead  to  a 
technique  for  objective  motion  measurements.  Several  sequences 
for  subjective  motion  evaluation  have  been  added.  These  sequences 
which  are  designed  to  demonstrate  codec  motion  performance  under 
high  stress  were  not  deemed  suitable  for  inclusion  in  the 
subjective  evaluation  tape  because  they  have  no  direct  relation  to 
material  that  may  be  expected  to  occur  during  an  actual 
teleconference.  The  duration  of  each  sequence  is  approximately 
one  minute  which  should  allow  sufficient  time  for  measurements 
without  producing  an  unduly  lengthy  tape. 
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SECTION  7  CONCLUSION  AND  RECOMMENDATIONS 


The  principal  output  of  this  task  consists  of  the  1"  video 
tape  which  will  be  used  for  subjective  evaluation  of  codecs.  As 
described  in  the  preceding  sections  of  this  report,  great  care  has 
been  taken  to  facilitate  an  impartial  and  dependable  evaluation  of 
the  resulting  codec  output  tapes.  In  addition  to  complying  with 
CCIR  and  CCITT  recommendations,  the  tape  represents  the  combined 
ideas  and  suggestions  of  many  experts  in  both  commercial  and 
military  and  civilian  government  teleconferencing.  Thus  all 
reasonable  steps  were  taken  to  cover  present  and  forseeable 
aspects  of  teleconferencing,  and  to  achieve  reliable  and 
meaningful  codec  test  results. 

The  test  signal  tape  is  a  secondary  output  of  this  task  and 
is  recommended  for  further  utilization.  At  present  the  only 
meaningful  evaluation  of  motion  codecs  is  subjective.  This  method 
is  cumbersome  and  costly.  It  would  be  highly  desirable  to  develop 
techniques  for  objective  measurements.  The  test  signal  tape 
contains  the  standard  signals  for  conventional  analog  tests.  It 
is  recommended  to  compare  the  results  of  the  subjective 
evaluations  with  the  objective  measurements  and  to  determine  the 
degree  of  correlation  between  them.  The  expected  result  is  a 
simplified  methodology  for  motion  codec  testing  which  may  reduce 
the  need  for  subjective  evaluation. 
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